fix: implemented graceful shutdown for workloadmanager#217
fix: implemented graceful shutdown for workloadmanager#217mahil-2040 wants to merge 4 commits intovolcano-sh:mainfrom
Conversation
Signed-off-by: Mahil Patel <mahilpatel0808@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Summary of ChangesHello @mahil-2040, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a graceful shutdown mechanism for the workloadmanager, which is a great improvement over the previous abrupt termination. The changes include adding a Close() method to the Store interface and its implementations, and a new Shutdown() method on the Server. My main feedback concerns a potential race condition in the shutdown order, where the store connection might be closed before background workers using it have terminated. I've left a detailed comment with a suggestion for a safer shutdown sequence.
There was a problem hiding this comment.
Pull request overview
This PR implements a comprehensive graceful shutdown mechanism for the workloadmanager component. Previously, process termination was abrupt—HTTP server shutdown was handled inline, but Redis/Valkey connections remained open until timeout, and background workers could be interrupted mid-operation. This change introduces a coordinated shutdown sequence where the HTTP server drains first, then store connections are closed, and finally background workers are signaled to stop via context cancellation.
Changes:
- Added
Close() errormethod to the Store interface and implemented it in both redisStore and valkeyStore - Replaced the inline shutdown goroutine in server.Start() with a dedicated Shutdown() method on the Server struct
- Updated main.go to call server.Shutdown() with a 15-second deadline instead of a 2-second sleep
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/store/interface.go | Added Close() method to Store interface for releasing resources |
| pkg/store/store_redis.go | Implemented Close() that calls redis client's Close() method |
| pkg/store/store_valkey.go | Implemented Close() that calls valkey client's Close() method |
| pkg/workloadmanager/server.go | Removed inline shutdown goroutine and added dedicated Shutdown() method |
| cmd/workload-manager/main.go | Updated signal handling to use server.Shutdown() with 15s timeout |
| pkg/router/session_manager_test.go | Updated fakeStoreClient to implement Close() method |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #217 +/- ##
==========================================
+ Coverage 35.60% 41.70% +6.09%
==========================================
Files 29 30 +1
Lines 2533 2611 +78
==========================================
+ Hits 902 1089 +187
+ Misses 1505 1390 -115
- Partials 126 132 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
… responsibilities Signed-off-by: Mahil Patel <mahilpatel0808@gmail.com>
|
@hzxuzhonghu PTAL! |
Signed-off-by: Mahil Patel <mahilpatel0808@gmail.com>
Signed-off-by: Mahil Patel <mahilpatel0808@gmail.com>
|
@hzxuzhonghu please check this! |
3 participants
What type of PR is this?
/kind enhancement
What this PR does / why we need it:
Implements a comprehensive graceful shutdown mechanism for the
workloadmanager.Previously, process termination was abrupt — persistent connections to the backend store (Redis/Valkey) remained open until timeout, and background workers could be interrupted mid-operation.
This PR introduces a coordinated shutdown sequence:
Server.Shutdown(ctx)Close()method on theStoreinterfaceChanges
Close() errorto theStoreinterface, implemented by both:redisStorevalkeyStoreShutdown(ctx)method to theServerstruct for coordinated shutdownmain.goto callserver.Shutdown()with a 15s deadline instead oftime.Sleep(2s)fakeStoreClientinsession_manager_test.goto satisfy the updated interfaceWhich issue(s) this PR fixes:
Fixes #215
Special notes for your reviewer:
The shutdown order is intentional:
This prevents requests from failing due to a prematurely closed store.
Does this PR introduce a user-facing change?: